DOCUMENT RESUME 



ED 325 613 



UD 027 810 



TITLE 

INSTITUTION 
SPONS AGENCY 
REPORT NO 
PUB DATE 
NOTE 



PUB TYPE 



Identifying Schools for Program Improvement and 

Conducting a Local Annual Review. 

Advanced Technology, Inc., Indianapolis» IK. 

Department of Education, Washington, DC. 

TAC-B-30 

90 

19p.; Materials presented at the United States 
Department of Education Chapter 1 Prograun Improvement 
Meetings — Regional Meeting (Detroit, MI, February 
12-13, 1990). 

Speeches/Conference Papers (150) — 
Legal/Legislative/Regulatory Materials (090) 



EDRS PRICE MFOl/PCOl Plus Postage. 

DESCRIPTORS *Annual Reports; Basic SKills; Compensatory 

Education; Elementary Secondary Education; *Federal 
Programs; *Measurement Objectives; *Measurement 
Techniques; Outcomes of Education; Preschool 
Education; *Progrcun Evaluation; *Pro9ram 
Improvement 

IDENTIFIERS *Education Consolidation Improvement Act Chapter 1 



ABSTRACT 

This document comprises information and various forms 
that can be used by schools to conduct an annual local review of 
their Education Consolidation and Improvement Act Chapter 1 School 
Program and identify prograuns needing improvement. The following 
materials are included: (1) a summary of requirements for a local 
annual review; (2) an outline of school-level information to be 
included in an eoinual review; (3) a description of the annual review 
process; (4) a form for reporting pre- and posttest scores (aggregate 
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effect of positive outlier, and the effect of negative outlier; and 
(12) an explanation of central tendency and variability as they apply 
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AGENDA 

U.S.D.E. Regional Meeting 
Detroit, Ml 
February 12-13, 1990 

• 

Identification of Schools for Program Improvement 

Federal Legislation Governing Identification of Schools 

Methods of Identification at District Level 

Evaluation Requirements for All Chapter 1 Programs 

Summary of Regulations for Local Annual Review 

Conducting a School-Level Annual Review 

Aggregate Performance: Mean or Median Data Bases 
Which Can Assist in Chapter 1 Evaluations 
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B. 



^""^^ C^h2«1^i"£SJ5S^ ^''"AL REVIEW 

OF CHAPTER 1 SCHOOL PROGRAM IMPROVEMENT 

Aggr»gatt Parformanot In Basic and Advancad Sidiis- AnnrAn»t<. 
required for every primary sut^ area. 

SJid SS^S?^"* in basic and advanced skills In reading, mathematics 

^ ^ 'comprehension' 

applications or aquivalert scores from nationally nomied math test. 

^' JV'rSSif'^Srr^ ^ arts or reading tests. 

ad!a^?klllf • -chlevement In both t)asic and more 

Exclude from analysis children who are not proficient In English. 
Measure student achievement over approximately 12 months. 
Report on either spring-to-spring or fail-to-fall testing Inten^al. 
Report in NCEs. 

Pjgrm in Daslrad Outcomaa: Show substantial progress In achieving 
1. Desired Outcomes are the LEA's goals to imorove the educatinnai 

Attain grade-level proficiency. 

Improve achievement in basic and mors advanced skills. 

^' J^rfcS5?2^ "^"^ ^ of aggregate 

3. LEA'S may use other indicators. 

WSTcSdfen^^r^ 
?a"rS^T2S3ned*r:r ^^-^ (^^gregate) 
^•asmant of parantal Im/olvamant program. (See P.L100-297. Sec. 1021 
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5. 
6. 
7. 
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CHAPTER 1 ANNUAL REVIEW 

Each school must conduct an annual review of data In 1 and 2, and possibly 
3-Q, and disseminate the Infomiation to parents, teachers, and others. 

Mml be discussed at annual review 

SchooNevel information 

1. Ag gragata gains for all educationally deprived students in grades 2 to 12, using 
Spring to Spring or Fall to Fan data. 

• Basic sidlls: record total reading and/or total math scores in 
NCEs 

• Advanced skills: record reading comprehension and/or mathematics 
problem solving and application scores in NCEs 

e Compute gains and losses and find the median or mean 

2. Monitor and assess desired outcomes at all grade levels using the criteria 
established for "Substantial progress." Set up databases by school for each 
outcome in your pro|ect application. 

May be discussed at annual review 

School-level infonnation 

3. Ijook at student level gains and at tainment of desired outcomes and make program 
modifications for cNldren who didn't gain. 

District-level infonnation 

4. Look at performance In the regular program. 

5. Conduct a sustained effects study every three years. 

6. Assess parent Involvement. 
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did you know that,, 

an annual review of each school's data must take place at 
the end of the school year or the following fall? 

The final regulations require that "for each project school" the LEA 
conduct an annual review of the effectiveness of the Chapter 1 
program. TTiis review must consider at a minimum the aggregate 
gains made on achievement tests by Chapter 1 students from Grade 
2 up, and (for the 1989-90 school year and beyond) the progress 
made toward attaining the desired outcomes stated in the LEA's 
appUcation. The annual review, which gcneraUy should be done in 

J^Sl.^' consider any evaluation inforaiation pertaining 

to the Pre-K, K, and Grade 1 components of the school's program. 

What is an annual review? Most likely it's an occasion when 
sever^ stakeholders sit down and review evaluation results together. 
The LEA Chapter 1 coordinator and the person completing the 
evaluation, for example, might meet with the building principal 
and Chapter 1 instractional staff. This would be a good time to 
identify children who didn't progress as expected and to start 
planning an appropriate course of action for both program and 
student improvement. Additionally, results of the review must be 
made available to teachers, parents of participants, and other 
appropriate parties. 

For schools that showed no improvement in aggregate perfonnance 
or did not show substantial progress towanl meeting their desired 
outcomes, a program improvement plan must be developed and 
unplemented. 

Documentation of the annual review for all Chapter 1 schools 
should be kept by the LEA and be available during SEA on-site 
monitoring. 
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student Identification 

A« part of the annual evaluation you are required to identify all students who have not shown 
substantial progress toward aeeting desired outcoaes or whose perfomance show no iaprovesent or a 



1. List all students in the Chapter 1 prograa. 

2 . Indicate their current grad* placeaent (during year when data was gathered) . 

3. Indicate a 1 for students not aeeting goals for the first tiae, and a 2 if this is the second (or 
■ore) consecutive year in %rhich the child was identified. 

4. Place (X)is to indicate areas in which the child did not iaprove. 
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did you know that... 

the median NCE gain may be used as a measure of aggregate 
performance for program improvement purposes? 

Federal regulations (Federal Regitter. May 18. 1069) atate that *...an LEA ahali Imptement 
[a program improvement plan] with respect to each school that . . . shows no 
improvement or a dedtne in aggregate perfbmnanoe of participating cNidren for a 12- 
nxxith period." 

According to the 'Draft Policy ManuaT (Octot)er. 1989). txrth the mean NCE gain and the 
median NCE gain may t)e used as indicators of aggregate performance Ibr program 
improvement purposes. The manual states that "Where SEAs approve the use of the 
median. LEAs must use them uNformly. e.g., an LEA must use either means or medians 
in an their Chapter 1 buildings~lhey may not be used selectively in individual schools to 
•elect the one which results in avoiding program improvement In a given building." 

The neasure of central tendency used by a school district should be the one which most 
accurately expresses the aggregate performance of Chapter 1 students. The mean, or 
average, NCE gain is determined arithmetically and is based on test scores of all students 
irrvolved in the program. The median NCE gain is the shgle score exactly in the middle 
of the distribution and separates the top 50% from the bottom 50% of students. 

Because the mean is based on all scores, it is affected by extreme scores which may 
inflate or deflate It disproportionately. For instance, in programs which have few students, 
if one or two students make very large NCE gains, the mean may be unnaturally large. 
In such a case, however, the median would not be affected and would therefore be a 
better measure of central tendency than the mean. 

When deciding which central tendency measure to use. consider the following: 

1. Sample Size • How many scores are being considered? As the sample size 
increases, the mean and median tend to become more similar. 

2. Tendency to have extreme scores - Are there special groups of students 
involved whose scores would tend to fluctuate more than most of the 
students in the group? 

3. Are computer fadltties available which will enable you to make necessary 
calculations? 

Will the median result in a valid representation of the effectiveness of the 
Chapter 1 programs in a miyority of the schools in the LEA? 

2601 Fortune Circle East • Indianapolis, IN 46241 
(317) 244-8160 • (800) 456-2380 January, 1990 
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An Exampit of Calculation off Aggragata Maan and Madlan NCE Scoras 

Assume that aach tat of scores depicted in Figures 1 through 3 represents 
p^fomjanoe of gn^tos 2. 3. and 4 in a school. Calculate me^n anTWan 
gam scores for the school as follows: 

Mean 



1. 



by tt» numbw or Mjdtnls wtwM toofw 
ww« UMdtooMMtthtinMn. Tlwn 
add bath tht numbw or ihidtnit (N) 

■cron tht gradM and thf product* (Total 
Qalni). 



Sradt 2 
CradiS 
•radi « 

Tatal 



s.o 

•2.« 



10 

11 

10 
SI 



•ilo 

ss 

•26 
41 



DMdo total oalna by tht totri numbar or 
itudanta In alihra* achooit. 

Tof 1 toliM ■ if] 

Tetat M ■ SI > 1.S2 



The schooMde mean NCE gain Is 1,32. 2 



3. 



Pit al of tha ftudam gain aooTM Into a fle 
and aoit tham from towaat to NghMt or 
ftom hlghaal to iowaat Nota that an 
oooroa from tha achod art rankad as a 

group. 

•31 
•28 
-11 
-7 
•e 

•4 
•4 

•2 
•2 
-1 
0 

1 11 

2 13 
2 13 
2 33 

a 



Count tha numbar or 
lharawasi. 



atudanta. In this 



Vtha numbar or midants a odd. count up 
worn tha bottom or down from tha top to 
tnd tha middto aoora. In tMa oaat, tha 

•Maanih aoora from tha top or bottom is 
3* 

IT tha numbar or atudanta Is mm, tha 
madlan Is haWwwy b atwaan tha two scoras 
art SKMiiy fr) tha middia or tha 
diitrlbUloa (Cxampia: A distribution 
oortWnathalolowIno acoras . 1, 2. 3. and 
4. Tha madlan Is hair way batwaan 2 and 
3. or 2.5.) 



The median NCE gain acore lor the school 
is 3. 



Nota: Matarial for this paper was adaptad from "Aggragate 
Parformanca Mtasuraa: Tha Maan or tha Madian" by Barbara 
Lawrancs/ Raaaarch and Training Asaociataa, Inc. 
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Figure 1: Little Difference Between Mean and Median 
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Figure 2: Effect of Positive Outlier 
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Figure 3: Effect of Negative Outlier 
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AGGREGATC PERFORMANCE MEASURES: 
THE MEAN OR THE MEDIAN 



CENTRAL TENDENCY 

The two most common descriptive measures of a group of scores are measures d 
central tendency and variability. That is, characteristics of a group can be described by 
idcnti^ing a middle reference point and summariziiu how the scores are arranged around 
it Different measures oi centra! tenden^ provide afferent information about the group. 
Two of IhWB, tibe median and the mean, may be used for aggregating data for purposes of 
assessing "aggregate performance." 

THE MEDIAN is the pointin a distribution of scores at which 50% of the scores fall 
above and 50% fall below-that B, the middle ranking number in a set of numbers. It 
identifies the central ranking score within a group without reference to the actual value of 
the score. This information is useful when k)oking at aggregate performance because it 
lessens the infhience of extreme scores in determining typical performance in the program. 

THE MEAN is determined arithmetically, and is based on the values of all scores in 
the distribution. It is what most people call the "average," and is the most commonly 
encountered measure of central tendency. The mean differs from the median in that the 
median is M central SCORE in the distributiou and the mean is the central VALUE of the 
distribution of scores. Eveiy score value in the distribution influences ibc mean. Therefore 
scores that are extreme cat atypical of the majority may inflate or deflate the mean, making 
it a less accurate indicator of typical performance of students in the program in those 
instances. 

VARUBILITy 

The amount of influence extreme or outlier scores will have on the mean also 
depends on the variability of the scores. Scores from a veiy large, heterogeneous group, 
if plotted on a continuum, will generally form the "beU curve" shape, with the majority of 
scores at the middle point on ^ continuum In this type of distribution, the mode, the 
median, and the mean have identioJ values. 

Scores based on smaller numbers and/or groups that are more homogeneous, 
however, are often not normally distributed-that is, the nugority of the scores are collected 
around a point somewhere above or bebw the middle point on the continuunL In this type 
of distributimi, the mode, the median, and the mean are different, and are distributed along 
the continuum in relation to where the majority of the scores cluster. If the cluster of scores 
is below the center of the continuum, the mean score vriU be hi^er than Hat median; if the 
duster of scores is above the center of the continuum, the mean will be below the median. 
The presence of one or two extreme scores at either end of the distribution can change a 
normal distribution with identical values for the mean and median to one where the mean 
and median are different. It is in these instances that consideration shovld be given to using 
the median instead of the mean for summarizing aggregate performance measures. 
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